Meta-Interpreter

summary

The meta-interpreter introduces a new way to call the expand-functions. Instead of directly calling the functions and processing the result, it will interpret a json-compatible description of the behavior. The advantages of this are:

• easier to write and add new languages / better readable language descriptions
• more error resistant description of new languages
• even the users can pass this descriptions as arguments and hence easily change existing languages or add own languages

motivation

In the current code base

• all expand functions are duck-typed

• and reused in the same way:

  result = expand(string, start, end)
  if result:
    return result

  Hence the calls could be done in a more generic way. In this pull request this is done with a meta-interpreter, which interprets a json-compatible python object. Hence it is possible to define languages easier and even on the user level possible.

meta-interpreter

expand-function: In the following the term "expand-function" is used for a function, which is currently used for every expansion. The signature is:

expand: (string: str, start: int, end: int) -> dict("start"->int, "end"->int, "string"->str, "type"->str)

This maps a selection limited by start and end in a string to a new selection or None.

The interpreter interprets a macro and returns the resulting object to expand. To do this there are two ways to register names in the interpreter:

1. register_command(name, function): registers a expand-function and binds it to a given name. This function will be called, when the name is interpreted.
2. create_macro(macro_name, macro): registers a macro (See Macro in syntax and semantics) and binds it to a given name. The macro will be interpreted if the name is interpreted. (This may create infinite recursions)

The user will be apple to call macros by names or directly write macros in the function call.

Developers can import the interpreter and register their functions or macros.
The interpreter must imported via a proxy and not directly, i.e. from ._minterp import interpreter instead of from .minterp import interpreter. This will import the interpreter module as a singleton.

syntax and semantics

This is an (informal) syntax definition with the corresponding semantical description.

• str is a python or JSON string ""
• List of is a python or JSON list [...]
• Dict is a python dict or a JSON object {"key": "value", ...}
• expand-function is a pointer to a expand-function (only python)

# the unit, which is interpreted
# the list of command will be searched until
# the first command with a result
Macro ::= List of Command | Command

# the possible ways to expand a selection to a result
# str: must be a valid name of a registered function
# expand-function: function pointer to a valid expand-function
#     (Not JSON compatible)
Command ::= str | expand-function | EnrichedCommand | ClosestCommand

# selects the closest command to the current selection
ClosestCommand ::= List of Command

# enriches the simple command with additional options
EnrichedCommand ::= Dict(
    # the executed command/macro
    command ::= Macro
    # (optional) restricts the string on which the command is executed
    [scope ::= Command]
    # (optional) will be forwarded to the expand function
    [args ::= Dict(str->object)]
    )

How to use:

• ClosestCommand: Just add a list of expansions and the closest one will be selected. If it is not definite, then the first one in the list will be selected, e.g. for parens and quotes in the text (Hello "W|or)ld".
• EnrichedCommand: Adds modifier to commands. Later additional useful modifiers might be added. Current modifiers:
  • scope: Limits the command on this scope, hence the string will limited to the result of the scope command.
  • args: Will be forwarded to the function. Therefore those args must arguments of the function, but the functions should define default arguments. A use-case could be the expand-regex function, which could accept a regex. E.g. {"command":"regex", "args": {"regex":"[\w']"}} could be used to expand to word characters and ' for match words like that's.

example

This creates the javascript language definition:

from . import interpreter
interpreter.create_macro("javascript", [
    "subword",
    "word",
    # search inside the quotes for parens/brackets
    {
        "scope": "quotes",
        "command": "symbol"
    },
    # search for the closest of line/quotes/symbol
    [
        "line",
        "quotes",
        "semantic_unit",
        "symbol"
    ]
])

impacts

I think it is a very easy and expandable way to define and add new language. Hence I would assume a lot of new languages and better maintainability of existing ones. In addition some user requests wishes for a better flexibility, which is provided by this pull request.
Since this is a major change I would change the Version number to v2.0.0.